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ABSTRACT 



The U.S. Department of Health and Human Services' Office of 
Planning, Research, and Evaluation at the Administration for Children and 
Families, and the Office of the Assistant Secretary for Planning and 
Evaluation, are working together with states and other groups to improve the 
measurement of child outcomes in state welfare evaluations and in other state 
data systems. The Project on State-Level Child Outcomes has two phases. The 
first phase is a 1-year planning and design phase; the second is an 
implementation phase for data collection, analysis, and reporting activities. 
This report summarizes a meeting to finalize the common core of measures for 
the states' evaluations, the second in a series of national- level meetings of 
the planning phase of the project. After providing a detailed agenda of the 
February 27-28, 1997 meeting, the report summarizes presentations and 
discussions in the following sessions: (1) recap from last meeting and update 

on related activities (Welfare Peer Assistance Network (WELPAN) meeting, 
states meeting, child care researchers' meeting, letters and proposals sent 
to funders) ; (2) update on states' evaluations and work on indicators; (3) 

working session on core constructs; (4) "From Constructs to Measures: 

'Filling the Notebook'" (what constitutes a "good" measure for use as an 
indicator or as an impact, criteria for selecting good measures, criteria for 
deciding if a measure should be in the common core or if it can be 
site- specif ic) ; and (5) child outcomes (discussion on which measures are to 
be used as indicators and which as impacts in the areas of child health and 
safety, cognitive achievement/educational attainment/school progress, and 
behavior and adjustment) . The report then summarizes the second day of the 
meeting. "Filling the Notebook" was continued, with discussion of intervening 
mechanisms/interim outcomes in: (1) child care; (2) maternal psychological 

well-being; (3) maternal attitudes and skills; (4) stress; (5) parenting and 
the home environment; (6) father involvement; (7) stability of living 
arrangements and family structure; (8) use of services/benefits; (9) income ; 
and (10) maternal education. Presentations were also given on possibilities 
for additional funding and next steps for technical assistance. The report's 
appendix summarizes a meeting convened to inform the child care components of 
state welfare waiver evaluations. (EV) 
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Project Overview : The U.S. Department of Health and Human Services ' Office of Planning, Research, and 
Evaluation at the Administration for Children and Families (ACF), and Office of the Assistant Secretary for 
Planning and Evaluation (ASPE) are working together with states and other groups to improve the measurement of 
child outcomes in state welfare evaluations and in other state data systems. ACF is providing grants to states 
instituting welfare reform demonstrations to augment their demonstration evaluations with measures of child 
outcomes and also to expand their data capability to track state-level indicators of child well-being on an ongoing 
basis. Under funding from ASPE and the other federal contributors and private foundations, the states are receiving 
technical support on these activities from leading researchers who are members of the NICHD Family and Child 
Well-Being Research Network. The Network’s technical support effort is led by Child Trends, Inc. 

The project has two phases. The first phase is a one-year planning and design phase which began October 1, 1996. 
The second phase will be an implementation phase for data collection, analysis, and reporting activities that will 
begin in the fall of 1997. Twelve states participated in the first phase: California, Connecticut, Florida, Illinois, 
Indiana, Iowa, Michigan, Minnesota, Ohio, Oregon, Vermont, and Virginia. 

This report summarizes a meeting on discussions around finalizing the common core of measures for the states’ 
evaluations. This meeting was the second in a series of national-level meetings of the planning phase of the Project 
on State-Level Child Outcomes. 

The project is sponsored by ACF and ASPE. Additional federal funding to support this project has been provided 
by the U.S ’ Department of Agriculture, the National Institute of Child Health and Human Development (NICHD), 
and the Centers for Disease Control. Several private foundations have contributed funding to support the 
organization of national level meetings, the provision of technical assistance to the states, and the preparation and 
dissemination of written products. These include: the Annie E. Casey Foundation , the Edna McConnell Clark 
Foundation , the George Gund Foundation , and the Smith Richardson Foundation. 

The following individuals have been involved in multiple aspects of this project: 

Administration for Children and Families Office of the Assistant Secretary for 

Howard Rolston Planning and Evaluation 

Alan Yaffe Martha Moorehouse 



Ann Segal 



Child Trends, Inc. 



Christopher Botsko 
Brett Brown 
M. Robin Dion 
Tawanda Greer 
Chisina Kapungu 
Sharon McGroder 
Suzanne Miller 
Kristin Moore 
Erin Oldham 
Martha Zaslow 



NICHD Family and Child Well-Being 
Research Network 
Jeanne Brooks-Gunn 
Natasha Cabrera 
Greg Duncan 
V. Jeffrey Evans 
Kristin Moore 



National Center for Children in Poverty 
Lawrence Aber 
Barbara Blum 



PROJECT ON STATE-LEVEL CHILD OUTCOMES: SECOND MEETING 

February 27-28, 1997 

Wyndham Bristol Hotel 
2430 Pennsylvania Avenue, N.W. 

Washington, D.C. 20037 
Ph: (202) 955-6400 / Fax: (202) 775-8489 



Thursday. February 27. 1997 



9:30-10:00 SIGN-IN; COFFEE/TEA 



1 0:00 INTRODUCTION: UPDATE ON PROJECT STATUS 



10 : 00 - 10:30 Welcome — Howard Rolston (Director, Office of Planning, Research, and 

Evaluation, ACF, U.S. DHHS); Martha Moorehouse (Office of the Assistant 
Secretary for Planning and Evaluation, U.S. DHHS) 



10 : 30 - 1 1:00 Recap from Last Meeting and Update on Related Activities -- Kristin Moore, 
Child Trends, Inc. 

Overview of topics covered at last conference 
Discussion of related activities 

o WELPAN meeting, Jan 23rd (IL, IN, IA, MI, MN, OH) 
o MDRC states meeting, Jan 29th (CT, FL, MN, VT) 
o Child care researchers’ meeting on Feb 7th 

o Other meetings? 

o Letters, proposals sent to funders 



1 1 :00-12:00 Update on States’ Evaluations and Work on Indicators — Representative from 
each state 



12:00-1 :00 LUNCH (will be provided) 

1 : 00-3:00 WORKING SESSION ON CORE CONSTRUCTS 

o Review CTI’s composite grid of child outcome (impacts and indicators) and intervening 
mechanism constructs and measures 



Thursday. February 27. 1997 (cont.t 



3:00-3:15 



<BREAK> 



3:15-3:30 FROM CONSTRUCTS TO MEASURES: “FILLING THE NOTEBOOK” 
o Brief presentation on what constitutes a “good” measure for use as an indicator, and what 
constitutes a “good” measure for use as an impact; criteria for selecting good measures 
o Criteria for deciding if a measure should be in the common core or if it can be site-specific 
(o See composite grid of constructs, for each age range) 

(o See one-page summaries of measures and rationales, by construct) 

3:30-5:45 CHILD OUTCOMES (specify which measures to be used as an indicators and/or 
which as impacts! 

3:30-4:15 1. Child Health and Safety 



4:15-5:00 



2. Cognitive Achievement/Educational Attainment/School Progress 



5:00-5:45 



3. Behavior and Adjustment 



5:45 



RECEPTION 
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Friday. February 28. 1997 



8:30 CONTINENTAL BREAKFAST 

9:00 “FILLING THE NOTEBOOK” (cont.) 

INTERVENING MECHANISMS/INTERTM OUTCOMES 
9 : 00 - 9:30 1. Child Care 



9 : 30 - 9:50 


2. Maternal Psychological Well-being 


9 : 50 - 10:10 


3. Maternal Attitudes and Skills 


10 : 10 - 10:30 


4. Stress 


10:30-10:45 


<BREAK> 


10 : 30 - 11:00 


5. Parenting and the Home Environment 


11 : 00 - 11:30 


6. Father Involvement 


11 : 30 - 12:00 


7. Stability of Living Arrangements and Familv Structure 


12:15-1:15 


LUNCH (will be provided) 


1 : 15 - 1:45 


8. Use of Services/Benefits 


1 : 45 - 2:00 


9. Income 


2 : 00 - 2:15 


10. Maternal Education 


2:15-2:45 


POSSIBILITIES FOR ADDITIONAL FUNDING - Howard Rolston; Barbara 
Blum 


2:45-3:15 


NEXT STEPS FOR TECHNICAL ASSISTANCE - Kris Moore 


3:15 


ADJOURN 



SUMMARY OF THURSDAY, FEBRUARY 27 PRESENTATIONS 



I. INTRODUCTION: UPDATE ON PROJECT STATUS 

Dr. Howard Rolston, Director, Office of Planning, Research, and Evaluation, ACF, U.S. 
Department of Health and Human Services, opened the meeting. Rolston commented that he 
was impressed with the amount of work that the project participants have accomplished thus far. 
He also acknowledged all of the state participants, Child Trends, and the other researchers who 
have contributed to the success of this project. Rolston announced that ACF had developed the 
criteria for the implementation phase of the Project on State-Level Child Outcomes. The criteria 
were distributed later in the meeting. He also noted that they might make modest changes in the 
criteria, depending upon the discussions at the meeting. Second, Rolston said that ACF was 
close to making the announcements for the Track 1 January 1 5 grants. Thirty states submitted 43 
applications for funding. Rolston noted that they were not able to fully anticipate what the Track 
2 awards would look like until after the Track 1 awards were announced. 

Dr. Martha Moorehouse, from the Office of the Assistant Secretary for Planning and Evaluation, 
U.S. DHHS, also provided opening remarks. Moorehouse said that she was also very excited 
about the work that the project participants have accomplished to date and about the future 
directions of this project. She also noted that the project has been conducted in a spirit of 
collaboration, and that this collaborative spirit should continue. Moorehouse provided thanks to 
the National Institute of Child Health and Human Development for their support of the February 
7 Child Care Researchers Meeting. She also acknowledged Dr. Martha Zaslow and Natasha 
Cabrera for their role in organizing this meeting. Moorehouse stated that much of the work to 
date on the Project on State-Level Child Outcomes has been focused on impacts designs. 
However, the Project has a dual interest in impact work and indicators work. Making 
improvements in states’ abilities to track trends in children’s well-being over time is an 
important part of the project. In order to spend more time focusing on the indicators area, DHHS 
is sponsoring a meeting on indicators on April 28. 

II. BACKGROUND AND UPDATE ON RELATED ACTIVITIES 

1. Conversations and meetings -- Kristin Moore, Child Trends; Bob Lovell 

Dr. Kristin A. Moore reported that, since the initial meeting in November of 1996, regular 
telephone conversations with state representatives have provided an opportunity to hear from the 
states and to provide information and assistance to them. States, meanwhile, were completing 
proposals to DHHS to support ongoing evaluation activities. 

One highly efficient mechanism for working with the states was suggested by Bob Lovell of 
Michigan and Tom Corbett of the Institute for Research on Poverty at the University of 
Wisconsin. Together with Theodora Ooms of the Family Impact Seminar, they have organized 
seven Upper Midwest states into an organization called WELPAN, the Welfare Peer Assistance 
Network. Since six of these states are members of the Project on State-Level Child Outcomes, it 
made sense to meet jointly with these states. In fact, the seventh Upper Midwest state, 

6 



O 

ERIC 



7 



Wisconsin, was invited to listen in for the day, and one evaluator, David Fein from Abt, was able 
to join the discussion as well. 

To assure that participants are communicating with common terms, the WELPAN meeting began 
with a review of the “Four I’s.” Even within the research community, the same terms are not 
always used. In order to be able to communicate across states, across research organizations, and 
across academic disciplines, a set of common terms are necessary: 

Indicators: a measure of a behavior or a condition or status that can be tracked over time, 
across people, and/or across geographic units. 

Impact Studies: experimental/control (E/C) studies in which participants are randomly 
assigned to either the experimental or the control group. If the study is well-designed and 
implemented, causal implications can be drawn for the population in the study. 

Intervening Mechanisms: the ways in which welfare programs may have impacts on 
children. Intervening mechanisms are first affected by a policy or program; they in turn affect 
children’s development and well-being. 

Inferential Studies: studies that fall between indicator and impact studies, that go beyond 
indicator studies in that they attempt to assess causality but which cannot provide definitive 
evidence regarding causality. 

Other terms that will be used over the life of this project were defined as well : 

Domain: a broad substantive topical area, such as “Child Health & Safety.” 

Construct: topics within a domain. For example, grade progression, cognitive achievement, 
and engagement in schoolwork are all constructs within the broader domain of educational 
attainment and cognitive achievement. 

Measure: an item or a set of items that provide a concrete way to assess a construct. For 
example, maternal depression can be assessed with a set of self-report paper and pencil 
questions asked of a mother. The individual items are combined into a depression scale. 

When a measure is included in an analysis, it may be described as a “variable,” e.g., our 
depression variable. 

Child Outcome Measure: an item, scale or index that assesses an aspect of the child’s 
development or well-being, e.g., health, school performance or behavior. (Measures like 
income and parenting are not child outcomes, but can be intervening mechanisms or 
underlying indicators of child well-being.) 

After reviewing these terms at the beginning of the WELPAN meeting, the group worked all day 
to select constructs within domains. Participants began with the lists developed at the November 
conference when state representatives met in small groups and generated a list of child outcomes 
and intervening mechanism; these lists were shared with states at the back of the summary of the 
first meeting. The goal for the WELPAN meeting was to go over this list and expand it. The 
technical assistance team wanted to make sure that the full range of potential outcomes being 
considered in the states would be represented in the list. The team wanted state representatives 
to really think through how welfare reform as implemented in their state might affect children, 
either directly or indirectly. 
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Thinking through this process sometimes led state representatives to think of unexpected effects 
(for example, that children in the experimental group might be more likely to arrive in school on 
time and be absent less often than children in the control group). Sometimes, this thought 
process led state participants to discard some expectations (for example, that work-oriented 
programs affect children's cognitive achievement or positive outcomes such as volunteering). 

Bob Lovell from Michigan acknowledged the support of the Joyce Foundation, the Institute for 
Research on Poverty and the Family Impact Seminar, and provided additional insights about the 
WELPAN meeting. Lovell said that the meeting participants’ goal was to “tell a story” about the 
mechanisms through which welfare reform will affect a family. The group considered whether a 
particular construct was a matter of public policy, whether it was related to welfare reform, and 
whether the construct was measurable. For the purposes of the WELPAN meeting, welfare 
reform was primarily focused on work. Meeting participants hypothesized that work would 
decrease parents’ isolation, routinize family life, improve parent-child interactions, and decrease 
parental stress. These intervening variables, in turn, will affect children. Lovell stated that the 
group decided we need to know about income (its form and source), characteristics of the parents 
(e.g., attitudes, skills, psychological well-being, involvement in drug and alcohol abuse), 
characteristics of the children (e.g., academic achievement, behavior, emotional well-being), and 
characteristics of the broader family and group (e.g., housing, role of grandparents). The group 
decided that in order to tell a complete story, it was necessary to consider variables beyond child 
outcomes. 

This conversation led to a discussion of priorities, which was completed by mail and which 
provides the basis for the current discussion. 

Kristin Moore commented on another issue that began to emerge during these meetings and 
conversations. This is the question of how indicators of child well-being fit into a project where 
child impacts have captured everyone’s imagination. From the beginning, this project has been 
designed as a way to help states develop or augment or make use of existing administrative data 
for their state. Administrative data can be linked with the records of the children in a state’s 
impact sample and be used for impact analyses. Less noted but of comparable importance is the 
fact that administrative data can also be used to report on the condition of all children in a state. 
In addition, other data resources that go beyond administrative data are available and can be used 
as indicators of the well-being of all children in the state. The importance of work on this topic 
was further emphasized at a dinner held that evening. Dr. Martha Moorehouse challenged state 
participants to link up with others in their state who are striving to assess child well-being 
beyond the welfare domain and to think about how information being collected about children 
and families in their state might be reported. 

3. Child care researchers’ meeting, February 7, 1997-- Martha Zaslow, Linda Brumleve 

Dr. Martha Zaslow gave a brief history of the meeting that was held on February 7, 1997, to 
inform the Project on State-Level Child Outcomes with regard to the choice of child care 
measures. She noted that child care had emerged as a high priority of the states in the November 
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meeting. A number of hypotheses were articulated for how state policies could affect child care, 
and in turn, children’s development. 

Child Trends staff thought that it would be productive to have a brief consultation from a 
research group at the National Institute of Child Health and Human Development (NICHD) 
working on the issue of child care. This research group, the NICHD Study of Early Child Care, 
has expertise in both recent research findings and measurement issues. Working together with 
Dr. Natasha Cabrera and Dr. Jeff Evans of NICHD, Child Trends started to plan for a brief 
workshop with some of the members of the NICHD Study of Early Child Care. 

However, it quickly surfaced that there was a great deal of interest in child care in the new policy 
context. At a planning meeting, a larger group of researchers and policy makers (at both the 
national and state levels) expressed an interest in expanding from a brief consultation to a one 
day meeting with formal presentations. It was noted that such a meeting could serve two 
purposes instead of one: (1) to help inform the choice of child care measures for the Project on 
State Level Child Outcomes, and also (2) to discuss what is known about child care for low 
income children, and what research is needed within the new policy context. 

Therefore, Cabrera and Evans collaborated with the team at Child Trends to plan a larger 
meeting. The meeting that was held on February 7th included about 50 participants and had 
about 12 presentations. These presentations focused on the new policy context, and also on 
research on child care used by low income families. A number of the people from the Project on 
State Level Child Outcomes were able to attend. 

Linda Brumleve from Illinois summarized her experience of the Child Care meeting as a state 
level representative interested in the issue. She noted that the meeting was productive and 
informative. Much could be gleaned about measures and measurement strategies for the 
purposes of the Project on State-Level Child Outcomes. The meeting was also extremely 
informative as to research findings. Linda Brumleve recommended that the state teams read the 
summary about the meeting that was prepared (the summary is appended to this document). 

4. Letters, proposals sent to funders 

A topic of considerable interest to state representatives concerns ongoing conversations with 
private foundations. Interest on the part of foundations in understanding how welfare reform 
affects children as well as adults has been strong , and only one of the multiple foundations 
contacted to date has said it would not be possible to provide funds for the project. Contact with 
foundations will be ongoing, in the hope that funds can be made available that will augment the 
funds that the government will be able to allocate to these studies. 

III. UPDATE ON STATES’ EVALUATIONS AND WORK ON INDICATORS 

Representatives from each state gave brief updates on their state’s activities related to their 
evaluations. In addition, state representatives described their plans for the upcoming May 3 1 
proposal. 
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1. California. Oshi Ruelas from California’s Department of Social Services provided an 
update of the state’s activities since the November meeting. Ruelas indicated that the first report 
for the California Work Pays program was released in December, and a report for the CAL Learn 
program is being prepared for release at this time. California submitted a proposal for both 
Track 1 and Track 2 funding in January. A variety of inter-agency meetings are taking place in 
California in an attempt to coordinate the linking of administrative data, and to explore which 
indicators will be most important to include for study. An interdepartmental meeting is planned 
for the child impacts study in which the team will explore what research questions should be 
asked, how to ask those questions, and to decide on an age group. 

2. Connecticut. Mark Heuschkel reported on his state's activities since the November meeting. 
Connecticut decided to continue their control group and keep their original waiver evaluation 
intact. Their evaluation includes an impact study and a cost-benefit analysis of JOBS First. 

They have also added some enhancements to the evaluation, including an 18 month survey. 

They also plan to increase their sample size from 1080 to 1700 for the 36 month survey so that 
they can do more subgroup analyses. Connecticut is also conducting a special study of clients 
who reach the 21 month time limit. The first group of clients will reach this time limit in 
October of this year. The study will include between 40 and 50 intensive case studies of 
families' work behavior, other sources of income, and changes in living arrangements. Families 
will be followed for one year. Representatives from Connecticut participated in a meeting in 
January of the four states that have Manpower Demonstration Research Corporation (MDRC) as 
their third-party evaluator. Since this meeting, the Connecticut team has been reviewing relevant 
literature and refining their hypotheses about the relationship between welfare reform and child 
outcomes. They have also been working with MDRC on finalizing their measures and 
evaluating the quality and accessibility of their administrative data. They have not yet decided 
how they will incorporate the development of indicators into this project. 

3. Florida. Don Winstead updated the group on Florida’s activities since the November 
meeting. MDRC has prepared a report on impacts for the Florida waiver experiment that should 
be released soon. The evaluation shows employment and earnings gains continuing over time. 
There has been a decline in welfare recipiency. The higher income disregard and other policy 
initiatives have not cost the state extra dollars. They are tracking administrative data and child 
care expenditures for the experimental and control group. There is a higher child care cost for 
the experimental group but not as high as might be expected. Florida is concentrating its 
continuing evaluation efforts on Escambia County, the county that has policies that are most 
similar to WAGES, the state’s welfare reform program. There are over 800 families that have 
been subject to the experiment for over 2 years and thus could have been subject to the time 
limit. However, a much smaller number have actually reached the time limit. There were a 
number of families who reached the final review process and withdrew in order to bank some 
months. More than 30 families have actually been terminated. Among these families, three out 
of 76 children have had benefits continued. MDRC is doing an extensive follow-up on those 
families that have hit the time limit. This includes in-depth qualitative interviews. Florida is 
maintaining old AFDC policies for the control group and submitted a Track 1 proposal. 
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4. Illinois. Dave Gruenenfelder gave an update of Illinois’ activities since the November 
meeting. Illinois will implement TANF in July of 1997. Consequently, the design of Illinois’ 
experimental study will change slightly. Illinois will now be concentrating on the study of time 
limits and work requirements. All program participants (e’s and c’s) will be equally subject to 
all other waiver and TANF provisions. The University of Chicago has been conducting 
interviews to determine the availability of data on children within the state. Although Illinois 
does not currently have a third party evaluator, they are planning on putting out a RFP for one 
soon. 

5. Indiana. Tracey Nixon provided a report on some of Indiana’s activities. They have been 
meeting with various agencies to develop a list of priorities for this project. They have been 
considering what is available, what can be collected and where the impact of their waiver 
demonstrations will be. The current priorities are access and use of health care and child care. 
They are unsure whether some of this will involve special studies. David Fein of Abt Associates 
provided some further updates concerning Indiana’s evaluation. Indiana submitted a Track 1 
proposal and is continuing to pursue its statewide experiment. The control group will continue to 
get traditional AFDC. The policies of the treatment group will change in May. This will extend 
the work requirement and time limit to most of those treatment group members who were 
previously exempt. This will enable them to make comparisons among groups that experience 
policy changes at different points in time. There will be a two year time limit that sanctions the 
adult’s contribution to the grant. They have a survey of 1500 clients that is currently in the field. 
The survey is collecting information on demographics, family structure, and some other 
information. Indiana will be issuing an early-impact report within a month or so which will 
cover those recipients exposed to 6-18 months of the original reforms. 

6. Iowa. Deb Bingaman updated meeting participants on events in Iowa. Iowa began welfare 
reform in 1993. They would like to continue their study, but without a control group. The study 
will include continuing their client surveys, some of which will be mixed mode (telephone and 
in-person). Particular topic areas that Iowa is interested in include spells of program use, 
recidivism, employment and its effect on the family, recipient earnings, and poverty rates in the 
state. In addition, the client surveys will include questions on family formation and stability, 
child outcomes, a client’s involvement in various programs, family circumstances and life 
events, attitudes, and clients’ impressions of TANF. Iowa is also continuing a series of focus 
groups which will include teenage parents, two-parent and single-parent families. Iowa is 
considering adding additional focus groups that may include legal immigrants, families with 
children less than three years of age, the unemployed, the sanctioned, and the exempt. The focus 
groups will be designed to understand the experiences and perceptions of recipients and their 
progress toward self-sufficiency. Perceptions of child care availability will be another 
component of the focus groups. 

7. Michigan. Bob Lovell summarized Michigan's activities. Michigan submitted six grants, 
including two related to their 1115 waivers and two related to state plans to evaluate welfare 
reform. Several major changes are going on within the state. As of February 18, 1997, Michigan 
has a new director of its welfare system. Beginning April 1, Michigan will be combining the day 
care worker, the employee trainer, and the case worker into one position called a Family 
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Employment Specialist. The state has also instituted an early retirement program which will 
have implications for the research efforts that Michigan has been conducting. Finally, Michigan 
will implement a new computer system in October. Michigan's evaluation of "To Strengthen 
Michigan Families" will be finished in April, 1997. In August of this year, the state will be 
hosting an evaluation conference. Participants, including policy makers, potential funders, and 
researchers from Michigan universities, will examine what is currently being done in Michigan 
to evaluate welfare reform programs and what future evaluations should be undertaken. 

8. Minnesota. Chuck Johnson presented an overview of Minnesota’s activities. MDRC, 
Minnesota’s third party evaluator, will be releasing an evaluation of Minnesota’s program, the 
Minnesota Family Independence Plan or (MFIP) this spring. The report will include 
employment and earnings impacts. Johnson noted that the impacts of MFIP on employment and 
earnings are large for long-term recipients. Minnesota will continue their control group. 
Minnesota has plans to field a 36 month follow-up survey, which will include a child outcomes 
component. The survey has been expanded by 25 minutes and it will be conducted in homes for 
a portion of the single parents in their sample. Johnson also said that Minnesota is conducting a 
large, comprehensive indicators project. The Department of Children, Families, and Learning 
has data on 60,000 school children across the state. 

9. Ohio. Jackie Martin updated meeting participants on Ohio’s activities. Ohio is considering 
adding child outcome measures to three evaluations. These include: Ohio First, LEAP, and 
LearnFare. For the evaluation of Ohio First, they are focusing on using indicators and 
conducting inferential studies as they dropped their control group. Ohio is developing an RFP in 
order to obtain a third-party evaluator of this program. LEAP is a program that offers financial 
incentives and sanctions for teenagers who are pregnant or parenting. MDRC has beeen working 
on the evaluation of this program and will release a report in June of 1997. Ohio has a waiver to 
continue LEAP through 2003. Thus, they will be able to follow the sample (including children of 
the teenagers) until they are in their late twenties. They will be developing an RFP this 
September to obtain a third-party evaluator to continue the evaluation of LEAP. Macro 
International is evaluating Learn Fare. The Ohio team has been working on many different 
program areas. Within the next two weeks after the second meeting of the Project on State-Level 
Child Outcomes, the team will develop their model of causal pathways between welfare reform 
and child outcomes. 

10. Oregon. Lily Sehon from the Adult and Family Services division presented the update for 
Oregon. Oregon did not submit any proposals under either Track 1 or Track 2 in January. In 
addition, Oregon plans to drop their control group and is not intending to apply for 
implementation funds on May 3 1 . They are interested in further discussions on indicators of 
children’s well-being. They have also expanded their indicator capacity by using the Shared 
Information System, an administrative data warehouse that involves several state agencies. 
Oregon has also developed other data processing and survey systems. The projects that they are 
conducting are guided by the Oregon Benchmarks, a statewide system set up in 1991 to provide 
state indicators in several different categories. 
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11. Vermont. Cheryl Mitchell from Vermont reported on the activities of the state since the last 
meeting. Vermont submitted a Track 1 proposal in January. They plan on maintaining their 
original experimental design for their waiver evaluation. They proposed adding 1000 people to 
their survey in the January proposal as well as adding an interim survey. Vermont continues to 
monitor their administrative data for measures of children in relation to an impact study as well 
as an indicator project. 

12. Virginia. Carol Baron from Virginia’s Department of Social Services reported on the 
activities of the state. Virginia’s VIP/VIEW program began in July of 1995, and for now, the 
experimental design is being maintained. On February 1 st, the mandatory TANF program was 
implemented, but the control group participants will be exempted from the work requirements 
and time limits until June 30, 1997. Virginia applied for Track 1 funding in order to complete 
analysis of the data that has been collected since the program began and to carry out a survey 
possibly containing some child outcomes. Virginia also applied for Track 2 funding for future 
indicators work and a TANF evaluation activity. Virginia Tech’s Center for Public 
Administration and Policy and Mathematica Policy Research have been retained by the state to 
assist in developing the May proposal. 

IV. SYNTHESIS ON PRIORITIZATION OF CHILD OUTCOME AND INTERVENING 
MECHANISM CONSTRUCTS 

1. Overall Conceptual Model, with Core Constructs — Martha Zaslow 

Dr. Martha Zaslow gave a presentation that traced how the steps made at the previous meeting of 
the Project on State-Level Child Outcomes, and in subsequent communications, have evolved 
into a conceptual model of how welfare policies could affect children. The steps that have been 
made thus far have involved two processes: (1) expansion, or seeking to articulate hypotheses 
and constructs in an inclusive manner; and (2) winnowing, that is, with the full set of possibilities 
before us, identifying those that the states view as most important. 

The expansion phase in articulating a conceptual model. At the November meeting of the 
Project on State-Level Child Outcomes, work towards a conceptual model began by identifying 
the three major components needed for a conceptual model (see Figure 1 , Initial Model, in 
Appendix,): 

• State policies 

• Intervening mechanisms 

• Child outcomes 

In the most basic terms, a model would need to specify the key components of the state welfare 
policy, identify the mechanisms or pathways by which such policies might affect children, and 
indicate which specific child outcomes might be affected. 

At the small group discussions at the end of the November meeting, the states made a great deal 
of progress in breaking these broad categories down into more specific ones. Three kinds of 
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intervening mechanisms were differentiated (see Figure 2, Differentiation of Intervening 
Mechanisms Following State Discussion in Appendix): 

• Aspects of adult life that were directly targeted by state welfare policies (for example, 
maternal employment and family income); 

• Other aspects of adult life that were not directly targeted by state welfare policies, but 
that nevertheless might be affected by these policies (for example, maternal psychological 
well-being); and 

• Aspects of the child’s environment that might be affected by the policy, because of 
changes brought about in adult life (for example, changes in the home environment that 
might occur when the mother is employed) 

In the same way, the states identified three types of child outcomes that they felt might be 
affected by their welfare policies, and that states would be most concerned about (see Figure 3, 
Differentiation of Child Outcomes Following State Discussions in Appendix): 

• Educational outcomes 

• Health and safety 

• Social and emotional adjustment 

In the small group meetings at the end of the November conference, the group leaders asked the 
state teams to articulate their specific questions and hypotheses for how state policies would 
activate intervening mechanisms that might, in turn, affect children’s development. In the 
ensuing discussions each team articulated a list of specific intervening mechanisms and child 
outcomes. For example, in these discussions, the teams went beyond identifying maternal 
employment as something that might be affected by their policies to specify that hours of 
employment and wages should be considered. The teams went beyond the broad category of 
educational outcomes to identify such specific aspects as expulsion from school and retention in 
grade as important. After the November meeting, staff at Child Trends compiled an inclusive list 
of the more specific constructs that the states had articulated. 

As a final step in the expansion phase, staff at Child Trends sent a summary list to the states, 
asking for feedback, and asked for the state teams to add any key constructs that appeared to be 
missing. 

The winnowing phase in articulating a conceptual model. The next step in the process 
involved prioritization. The inclusive list of intervening mechanisms and child outcomes was 
revised in light of feedback from the states. Each team was asked to identify their highest 
priorities (see copy of form sent to states for prioritization in the Appendix). The small group 
meetings that have occurred since the November meeting also focused on the key task of 
prioritization. 

Dr. Zaslow then presented an overview of the conceptual framework that emerged from the 
prioritization process (Figure 4, Conceptual Framework Following State Discussions in the 
Appendix/ For each of the constructs (boxes) in the conceptual framework, Zaslow also 
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reviewed the specific aspects that states had identified as important to measure (see Figures 5-8 
identifying Aspects of Constructs to be Measured Following State Prioritizations). 

It was noted that the model is primarily a “left to right” model. That is the boxes on the left in 
the figure are assumed to progressively lead to the boxes on the right, with state policies 
affecting aspects of adult life that are directly targeted by the policies as well as other aspects of 
adult life; these in turn are expected to affect children’s experiences in their primary care 
environments and subsequently child outcomes. Zaslow noted, however, that more complex 
feedback loops were possible. For example, the child care environment could have implications 
for the mother’s employment. The lack of arrows showing specific lines of causation in the 
conceptual framework figure was noted. Arrows are absent at this point because each of the 
states will vary in exactly how they will want to fill in the arrows, according to their states’ 
policies and specific hypotheses. She also noted that the conceptual framework would provide 
the basis for choosing measures for the “common core,” or set of measures to be used in common 
across each of the child outcome studies. States could choose to add further constructs and 
variables if they wished to go beyond the common core in their evaluation studies. 

Zaslow then asked the state teams for discussion about the conceptual model to this point, asking 
specifically whether the teams saw a need to revise at this point. There was an extremely helpful 
discussion. It was noted that the identification of constructs and specific variables within these 
models should allow for program impacts in a positive as well as negative direction, whereas the 
wording used for certain of the constructs seemed to anticipate only negative outcomes. Thus, 
for example, the states requested that the term “marital birth” as well as “nonmarital birth” be 
used within the construct of family formation. There was a discussion of the likelihood of 
maternal depression changing as a result of state policies. The possibility was raised that states 
might want to attempt to measure such positive aspects of maternal psychological well-being as 
life skills or sense of optimism. 

Participants looked at the conceptual model also from the perspective of constructs that might be 
missing. For example, it was noted that there was no construct seeking to measure neighborhood 
or housing quality. Yet families may move if their incomes change as a result of the state 
policies, and their residential situations may change (for better or for worse) accordingly. 
Participants at the meeting noted the importance of the issue of flexibility in work and child care, 
which did not yet seem to be addressed in the present conceptual model. There was discussion of 
the importance of attempting to measure premature employment in older children in the sample, 
and of levels of family income that might need to be passed before child outcomes were affected. 

Having examined the conceptual model for missing constructs and variables, the group then also 
noted the risk that a common core of measures, based on the conceptual model, was overly long. 
The discussion raised the possibility that we may need to do some further prioritization. One 
possibility that was raised was that of focusing on aspects of adult life that are directly targeted 
by the policies, but not on the other aspects of adult life that, while not directly targeted, may still 
be affected by state policies. 
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2. Subgroup Analyses— Kristin Moore 

Kristin Moore noted that states need to be concerned not only about constructs and measures, but 
the design of their studies as well. One issue that requires advance planning is the implications 
of subgroup analyses for sample design. Subgroup analyses require a large number of cases. For 
example, analysts might wish to examine impacts separately for long-term versus short-term 
recipients. Looking several years into the future, it may be the case that overall child impacts are 
found. Whether positive or negative, policy makers are likely to want to know whether such 
impacts occur across their entire client populations or are concentrated within particular 
subgroups. For example, impacts might be found among long-term recipients but not short-term 
recipients. Alternatively, impacts might be found among rural but not urban recipients. On the 
other hand, no overall impacts might be found. However, when subgroups are examined it may 
be the case that impacts are found. For example, impacts may occur among recipients with high 
levels of depressive symptoms but not in the larger population of recipients with no or few 
depressive symptoms. 

Available research and theory suggest that the implications of welfare reform for children are 
quite likely to differ for different groups of children, and it will be important to plan ahead to be 
able to conduct subgroup analyses. Power analyses need to be consulted to ascertain whether 
planned sample sizes are sufficient to detect likely impacts for the total sample and for subgroups 
of the sample. 

V. A DISCUSSION OF WORKING PARAMETERS-Martha Zaslow and Howard 
Rolston 

Howard Rolston noted the need for common working parameters as we move towards a common 
core for this project. While states will need to make some decisions separately, there is also a 
need to be in agreement across the states for certain common decisions. 

Dr. Martha Zaslow then led a discussion of a preliminary list of such “working parameters.” A 
handout (that had been reviewed by government project officers) was distributed as a starting 
point for a discussion on working parameters. The state teams reviewed and discussed the 
following elements of the working parameters document: 

• Suggested measures for the common core are based on the constructs identified in the 
prioritization process. The technical assistance team has assembled options for the common 
core measures. The team has also provided suggestions for measures for those constructs that 
the states identified as important at the November meeting and then that “survived” the 
process of prioritization. 

• Comparability across states. The purpose of the common core is to come up with measures 
that can be used across states, and thus provide some comparability of data. 

• Flexibility for states. It is assumed that states can opt to go beyond the measures of the 
common core, for example by carrying out in-depth studies of special issues (e.g., on-site 
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observations of child care settings), or by requesting funds to improve their state-level 
indicators data. It is assumed that states proposing to carry out augmented studies of particular 
topics will also request additional funding for such studies. 

• It is assumed that states can opt to drop sections of the common core, for example if the issues 
covered have already been addressed in previous survey waves for the sample. States may 
also opt to drop sections of the common core if they provide good justification that an 
intervening mechanism or child outcome is not likely to be affected by state policies. 

• It is assumed that states can opt to substitute administrative data for the survey measures being 
proposed, with appropriate justification and assurance of data quality. 

• Mode of data collection. In assembling a set of suggested measures for the common core, the 
technical assistance team assumed mixed mode administration of a survey, that is, telephone 
administration with in-home follow-up of families who could not be reached by phone. It is 
important to note that all measures that can be administered by phone are also appropriate for 
in-home administration. A key decision for the group is that of whether a survey approach 
will be taken and if so, whether the states will use mixed mode or in-home administration. 

• Focal child. The technical assistance team assembled possible measures for each construct 
assuming that a focal child would be selected in each family. However, some questions may 
be asked about all children in the family, and thus suggested measures are marked as “focal 
child” or “all child” measures. 

• Child age. In assembling possible measures, the technical assistance team assumed that states 
may choose to focus on specific, delimited age ranges in carrying out the core study (for 
example, elementary school age or adolescents) rather than focusing on a wide age range. 

Thus, the age ranges for which child outcome measures are appropriate are noted. In addition, 
specific augmented studies may choose to focus on a very narrow age range (for example, a 
state might opt to focus on the issue of infant day care, and accordingly select measures 
appropriate for this delimited age group). A key issue for the group to discuss is that of 
whether any limits should be placed on child age for the state studies. 

Discussion and Decisions. An open discussion followed the review of these working 
parameters. The discussion focused on the key decisions of (1) whether states needed to field 
surveys for their child outcomes studies or could rely primarily on administrative data, and (2) 
the range of child age that the present studies should focus on. Please note that some of the 
following decisions were finalized following post-meeting discussions between CTI and HHS. 

After extensive discussion, it was agreed that in order to have comparable data, the states 
examining child outcomes should all plan to field surveys. The working parameters regarding 
substitution and deletion will, however, hold. If a state has already collected key information in 
previous surveys, it will not be necessary to do so again. States may propose the substitution of 
administrative data for selected constructs in the survey (though not the full survey) with 
appropriate justification and assurance of data quality. 
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The advantages of in-home as opposed to telephone surveys were noted. These include the 
greater ease of collecting child care histories and the greater reliability of the data when mother 
and interviewer use a calendar format to record the information; the ability to collect direct 
assessments of children’s cognitive development; and the ability to complete ratings of the home 
environment and housing circumstances. 

The discussion concerning child age resulted in the decision that the present child outcome 
studies would focus on children of 5-12 years of age. This decision was based on the fact that 
the child outcome studies, in most instances, will be fielded a period of years after random 
assignment to waiver evaluations. This age range would thus permit consideration of child 
outcomes for children who were preschoolers when their mothers became subject to waiver 
policies, and children who were of school-age already at this point in time. In addition, states are 
interested in considering outcomes for school-age children, where special services, both in school 
and out of school, may begin to be involved for children experiencing difficulties. Only a small 
number of state proposals had indicated plans to study adolescents. Studies of adolescents would 
still be possible as augmented studies. However, the group decided that the common core would 
involve school-age children. 

The team affirmed the decision to select a focal child within this age range within each family, 
but also to ask certain questions for all children in the family. 

VI. FROM CONSTRUCTS TO MEASURES: “FILLING THE NOTEBOOK” 

A number of sessions then focussed on the very practical task of examining measures and 
discussing the rationales for including constructs and specific measures. 

Intervening Mechanisms/Interim Outcomes: Targets of Welfare Policies 

1. Income 

Many welfare policies are designed to change income. Welfare reform may change income 
because: 

• employment leads to higher income; 

• increased income disregards may lead to an increase in family incomes; 

• sanctions may produce a decrease in family income; 

• greater emphasis on employment may mean a greater percentage of income comes from work 
rather than welfare; 

• the unsteady nature of low-skill employment may lead to greater instability in income. 
Changes in income may in turn affect children because: 

• increased income may in turn allow the parent to purchase better quality child care, food, 
health care, and educational resources; 
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• decreased income may increase parental stress levels and decrease the resources available for 
the child; 

• declining or unstable income may make it difficult to pay the rent and this may lead the family 
to move or double-up; alternatively, higher income may enable the family to move to a 
neighborhood that is safer or has better schools. 

Possible Measures. The winnowing process began with a list of constructs that included total 
income, sources of income, and stability of income. Various measures were reviewed and a 
debate ensued over the usefulness of administrative data. Some states thought they could make 
greater use of administrative data; others were less sure. Many participants expressed a concern 
over the potential length of the income section. There was a suggestion that financial strain and 
material hardship would be worth considering for inclusion. In response to the group’s concerns 
over length, Child Trends decided they would confer with Greg Duncan (who wrote the income 
rationale), look at the Minnesota MFIP questionnaire and the Indiana questionnaire and present a 
more limited number of items that would get at the crucial areas regarding income. 

2. Employment 

Employment is one of the key targets of welfare reform. Included among the ways that welfare 
reform may affect employment are: 

• stricter work requirements may lead to higher and/or longer hours of employment; 

• work requirements may lead more and more parents to take jobs they might have passed up. 
This may lead to an increase in jobs characterized by shift work and “off-hours” work; 

• training programs may result in improved soft skills and job retention; 

• work requirements may result in more ill-prepared individuals entering the labor force. They 
may be unable to retain jobs for any length of time. 

Some of the reasons changes in employment-related behavior may affect children include: 

• successful employment may lead the parent to enforce routines. These routines may result in 
the child performing better in school; 

• some parents may fail to get a job or end up getting fired; this may result in depression with 
negative effects for the child; 

• steady employment may lead the parent to find a stable, high-quality child care situation that 
has a positive effect on the child’s school readiness or school performance; 

• a parent with an “off-hours” job may place a child in an inadequate, changing or ad hoc child 
care situations, producing problems for child’s health, socioemotional development, and 
safety. 

Possible Measures. The group considered the initial list of employment constructs and 
examined some of the question items designed to get at those constructs. Once again there was 
an extensive debate over administrative data and the potential length of any survey section on 
employment. A decision was made that it would not be useful to measure soft skills given the 
quality of existing measures. There was a concern that any measure of employment needed to 
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get at whether a person held multiple jobs at the same time. As with income, Child Trends 
decided they would confer with Greg Duncan (who wrote the income rationale), look at the 
Minnesota MFIP questionnaire and the Indiana questionnaire and present a more limited number 
of items that would get at the crucial areas regarding employment. 

3. Family Formation and Dissolution 

Family formation, dissolution and non-marital childbearing did not suddenly emerge for the first 
time as issues in TANF. Indeed, a number of the state waivers over the years have had among 
their goals reductions in non-marital childbearing, reductions in family dissolution, and increases 
in the formation of 2-parent families. Expectations are that: 

• work requirements, time limits, sanctions and/or a family cap may reduce non-marital 
childbearing; 

• stronger child support enforcement may discourage parenthood outside of marriage; 

• time limits and sanctions may cause families to double up, either with grandparents and 
relatives, or with unrelated individuals; 

• families may form or stay together despite high conflict, due to economic pressures. 

Changes in family structure are anticipated to have a number of implications for children in turn. 
In fact, a considerable body of research suggests that two-parent families would foster child 
development, particularly if the parents are the child's own biological parents and it is a low- 
conflict marriage. Specifically: 

• smaller families would enhance child development; 

• two-parent families would have higher income, and stronger investment in the child such as 
monitoring and routines, especially if families are formed by biological parents; 

• high conflict marriages/unions may undermine children’s development; 

• living in a supportive situation, perhaps with grandparents, may be supportive for parents and 
children; 

• doubled-up households may be crowded and increase the risk of abuse or violence. 

Possible Measures. The technical assistance team suggested covering issues of family 
formation and dissolution using a set of questions pertaining to changes in living situation similar 
to those used in the New Hope project. These questions would have to be modified for use on 
the telephone, but the general format could be preserved. A person could be asked if they have 
experienced a marriage, separation, or divorce since random assignment. If they replied “yes” 
then they could be asked when this occurred. Asking when the change occurred is useful 
information in itself and is likely to improve the accuracy of reporting. 

4. Changes in Attitudes Toward Work and Welfare 

Welfare reform was hypothesized to affect recipients’ attitudes toward work and welfare in the 
following ways: 
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• work and training experiences will produce more positive attitudes about work among parents; 

• general discussions about welfare and work will produce increased expectations for work, 
more positive attitudes about work and marriage, and less positive attitudes about welfare. 

Attitudes toward work and welfare, in turn, were hypothesized to affect children’s well-being in 
the following manner: 

• positive parental attitudes about work and/or negative attitudes toward welfare will increase 
children’s educational goals, school engagement, and school success. 

Possible Measures. Seven items from the National Survey of American Families which tap 
mothers’ opinions on the effects of welfare on other life decisions (e.g., finding work; having a 
child) and attitudes about maternal employment were recommended. This survey is still in the 
field, hence no reliability or predictive validity information in available. Also recommended 
were eight items from the Five-Year-In Home Interview of the JOBS Child Outcomes Study. 
These items are related to mothers’ employment and welfare and the perceived barriers to their 
becoming employed. After discussing this domain at the meeting, state representatives preferred 
not to include measures of attitudes in the common core. 

INTERVENING MECHANISMS/INTERIM OUTCOMES: Other Factors that May be 
Affected by Welfare Policies 

5. Resident Parent’s Psychological Well-being 

At the November 1 996 meeting, states tended to focus their interest in parental psychological 
well-being on one key aspect: maternal depression. Discussion of depression started by defining 
it as a negative mood state so extreme that it interferes with daily functioning and productive 
activity. Previous research was cited, showing that depression rates in samples of low-income 
women with young children range from 42% (in the JOBS Descriptive Study) to 60%. In 
thinking about whether welfare reform policies may increase or decrease maternal depressive 
symptoms, we hypothesized that: 

• working in low-paying, unstimulating jobs, or hitting time limits or sanctions may increase 
depression; or 

• becoming self-sufficient may put mothers into a more positive frame of mind and decrease 
symptomatology. 

It was pointed out that high levels of depression and stress have been found to lead to parenting 
that is more harsh, hostile, and less emotionally supportive to the child. A variety of child 
development studies have established that children of depressed parents: 

• display higher levels of behavior problems, both externalizing behaviors, such as aggression, 
and internalizing behaviors, such as anxiety and depression; 

• have been found to have deficits in social and academic competence; 

• often have poorer physical health 
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The CES-D, a widely used survey measure of depressive symptoms, was recommended. The full 
20-item version of the scale has high reliability and validity and has been used in other welfare 
program evaluations. For shorter surveys, or for telephone formats, shorter versions are 
available, though established cutoff scores are not yet established for these. A telephone version 
of the CES-D is planned for the parent questionnaire of a large-scale nationally representative 
survey of early childhood education. 

Some participants questioned whether depression is sensitive to the effects of programs and 
whether it is the most appropriate construct to measure in the domain of parental psychological 
well-being. However, other participants noted the substantial variation in depression across 
persons in varied life circumstances and others felt it was important to rely on the knowledge that 
has already been established regarding the link between parental depression and children’s 
outcomes. 

6. Absent Parent Involvement 

In previous discussions with the state representatives, many people have said that the core 
constructs should be focused not just on the resident parent, but also on the nonresident parent. 
Thus, this section is entitled "Absent Parent Involvement." However, since in most cases, state 
waiver provisions concerning nonresidential parents are targeted toward fathers, the hypotheses 
and measures that were presented were focused on fathers. 

First, hypotheses on the relationship between welfare reform provisions and absent parent 
involvement were presented: 

• enforcement of child support and paternity establishment may increase contact between 
nonresident parents and their children; 

• forcing fathers to acknowledge their paternity and pay child support might increase feelings of 
anger and resentment toward both the mother and the child; 

• enforcement of formal child support payments may affect the nonresident parent’s provision of 
informal support, such as the provision of cash, groceries, clothes or diapers, to the child, or 
affect the involvement of the father’s family with the child. 

Then, hypotheses on the relationship between absent parent involvement and child outcomes, in 
general, were presented: 

• increased involvement of absent parents in the form of child support payments is anticipated to 
have positive implications for children’s well-being, particularly on cognitive development, 
academic achievement, and behavior problems; 

• alternatively, increased stress and conflict between parents, if child support enforcement may 
have a detrimental effect on children’s well-being if the quality of the father-child relationship 
is poor or inconsistent or the relationship ends. 

Possible Measures. Administrative Data : States are already collecting administrative data on 
child support and paternity establishment. The question is whether for the states’ evaluations, 
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they will be able to link data from the child support enforcement agencies to their research 
samples. Data quality is another important consideration. In addition, administrative data will 
not capture the provision of informal support—either economic or noneconomic--or the amount 
of contact that a father has with his child. 

Survey Data : Meeting participants were provided with several examples of child support 
modules which include questions about formal child support payments, informal child support 
payments, and other aspects of informal support such as purchasing toys or groceries. In 
addition, they were provided with examples of questions about father-child contact— both in 
person and through letters and phone contact. These include questions from the JOBS 
Descriptive Study; the Child Development Supplement of the PSID, the National Survey of 
America’s Families, and the child support topical module from the Survey of Income and 
Program Participation. 

7. Stability and Turbulence 

Turbulence was defined as multiple changes in life circumstances. There are a number of ways 
that welfare reform may alter the level of stability or turbulence in a child’s life. Examples 
include: 

• parents who become employed may use child care assistance to enter into a stable, long-term 
child care situation; 

• sanctioned parents may be unable to pay rent and end up moving between relatives and 
friends; 

• children may change schools and/or child care arrangements frequently; 

• employment requirements may lead parents to acquire a steady job with a stable income; 

• employment requirement may result in a parent taking a job that requires shift-work. Such 
parents may have to rely on multiple and shifting child care arrangements. 

Changes in the level of stability and turbulence have been shown to affect a number of different 
aspects of a child’s life. Examples of how this may happen include: 

• employed parents may locate a stable, high-quality child care situation, which can improve the 
child’s school readiness and reduce behavior problems; 

• a chaotic and frequently changing living situation may result in a deterioration in child’s 
school work; 

• lower income may make the family move and the child may then adapt poorly to a new school 
resulting in an increase in behavioral problems. 

Possible Measures. The initial list of constructs was presented along with a set of measures. 
Most of the group agreed that this construct is closely linked to changes in welfare policy. There 
was a strong sense that both stability and turbulence needed to be covered. Participants stressed 
the importance of focusing on the child’s experience since the child may be separated from the 
parent. 




23 



24 



8. Consumption 



Changes in welfare policy will potentially alter family consumption patterns. Among the ways 
this might happen are the following: 

• increased income from higher income disregards may decrease the percent of family income 
needed for housing and food, making more money available for other purposes; 

• sanctions may reduce income, making it difficult for families to continue to afford their 
current residence; 

• work requirements may result in children being enrolled in a day care center where they are 
served a better breakfast and lunch than they were eating previously; 

• family income may decline and the family may end up spending less on food and as a result 
may have a poorer quality diet. 

Some examples of how changes in consumption may affect children include: 

• increased income may enable a family to move to better housing while spending the same 
percentage of their income on housing; 

• decreased income may mean housing requires more of the family budget and eventually may 
result in families moving in with friends or relatives; 

• sanctions may result in decreased income, squeezing the family’s food budget, but families 
may be able to make it up by an increasing reliance on food banks. 

Possible Measures. Three areas of consumption were discussed. These included percent of 
income spent on child care, housing and food. It was stressed that any such measure needed to 
include concurrent measures of both income and the area of consumption being examined. Child 
care measures had been discussed under child care. Concerns were raised over the difficulty of 
getting strong measures of food consumption. Steven Carlson of the Food and Consumer 
Service in the U.S. Department of Agriculture suggested that states interested in the topic 
consider a set of questions they have developed as part of the food security measurement project. 
This will be discussed in the summary of the discussion of health and safety outcomes. Concerns 
were raised about measures of housing consumption since it is difficult to get a clear reading on 
housing-related costs such as utilities. There were suggestions that measures of financial strain 
may be a useful way of getting at some of the same concerns which led to the inclusion of 
measures of consumption. The group decided to retain “% of income spent on child care and 
rent” as part of the common core. 

9. Use of Health and Human Services 

At the November meeting, states identified the use of health and human services as an important 
domain for measurement. Mothers who are mandated to participate in employment and 
educational activities through the welfare office might also be introduced to additional resources 
available to them. Welfare reform may affect participants’ use of health and human services in 
the following ways: 
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• case management through the welfare office may identify families’ service needs and improve 
access to needed services; 

• referral to needed services may facilitate parents’ leaving welfare for employment (e.g., child 
care subsidies, Medicaid, Food Stamps). 

In addition to supporting the mothers’ transition from welfare to work, these services are likely to 
benefit children in the following ways: 

• Food Stamps may allow the family to purchase additional/more nutritious food, improving 
child health; 

• quality child care benefits children’s health, educational, and socioemotional development; 

• Medicaid can ensure proper medical care when needed, improving children’s health. 

Possible Measures. Data for the use of health and human services are often collected 
administratively. However, such data may sometimes be incomplete or have other quality 
problems, or identified records may be difficult to obtain for a sample. In addition, 
administrative data may not be informative of whether the child may have health care coverage 
other than Medicaid (such as private insurance or an HMO), or whether the family is having 
difficulty accessing services. Thus, some options for mixed mode or in-home surveys were 
presented. Meeting participants suggested that other health and human services be added to the 
list, such as the use of mental health services, alcohol and drug services, Child Protective 
Services, and the use of heating and energy assistance programs. Meeting participants, however, 
decided to retain the set of core constructs (Food Stamps, Medicaid, childcare subsidy, access to 
medical care). States could still measure use of other services, but these would not be part of the 
common core. 
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SUMMARY OF FRIDAY, FEBRUARY 28 PRESENTATIONS 



I. INTERVENING MECHANISMS/INTERIM OUTCOMES: Child’s Environments 
Likely to Be Affected by Previous Intervening Mechanisms 

10. Child Care 

Child care is of great importance to the evaluation of welfare waivers and welfare reform as it 
is generally crucial for parents’ success in employment and self-sufficiency activities. Child care 
is typically defined as any non-matemal or non-parental care arrangement. Types of child care 
include child care centers, family day care homes, relative or neighbor care in or out of the 
parent’s home, Head Start programs, and before or after school care. Once the child is of school- 
age, the notion of child care is expanded to include lessons, regular activities, and self-care. 

For the possible effects of welfare reform on child care, it was hypothesized that: 

• increased work participation leads to increased use of child care; 

• increased child care subsidies and transitional benefits may lead to increased use of child care 
and use of child care over longer periods; 

• reliable child care may contribute to sustained employment; 

• increased demand for care may lead to increased use of unregulated care. 

Regarding the effects of child care on children’s development, there is evidence that participation 
in child care can lead to: 

• enhanced cognitive development especially for low income children when the child care 
environment provides a stimulating environment from which to learn; 

• enhanced socioemotional development through the child’s interactions with other children and 
with competent, sensitive teachers. 

On the other hand, if the child care arrangement is of low quality then the child’s development 
might be put at-risk in terms of both their cognitive and social development. For instance, high 
rates of teacher turnover have been shown to be related to aggressive behavior. Alternatively, 
then: 

• low quality or inconsistent arrangements may undermine children’s development. 

Possible Measures. Administrative Data . Administrative data could be used to measure some 
of the aspects of child care that states might want to capture. Administrative records could be 
used to assess the availability of care, use of subsidies for care and the cost of care, although the 
cost of care is most likely not broken down by child. A particular state may have additional child 
care information available. It is very possible that administrative data may be very helpful in the 
future due to new reporting requirements. Currently, administrative data are restricted to those 
who receive services and subsidies. 
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Survey Data: With regard to survey data, a number of possibilities were considered. State picks 
for child care included accessibility of care, availability of care especially for nontraditional work 
hours, infant care and sick care, cost of care, and type of care. The technical assistance team 
recommended that the states also measure the extent of care. This allows states to know how 
much time the child is spending in the care situation. After much discussion about whether the 
states agreed with the initial picks, we came to consensus that the focus of the child care 
measures should be the child. Therefore, our core list changed to the following: 

a question on the type of care 

a question on the number of children in group (group size) 
a question on the number of teachers with children in group (ratio) 
a three item quality scale from Arthur Emelin 

questions on stability (number of care givers over past year, number of care arrangement using 
during last week, scheduling of arrangements-on a weekly basis or same from week to week) 
a question on extent of care (number of hours in care per week) 

Beyond the key questions for all of the states, we also decided on a number of child care 
questions for states that want to focus in particular on child care in their evaluations. This second 
group of questions would include: 

a question on relation of child care to work (number of days missed or late because of child 
care) 

a question on availability of sick care and care in general 
a question on reliability of care (Emelin’s three item scale) 

11. Home Environment and Parenting Practices 

The home environment is another proximal environment that will directly affect the child. 
Parenting practices represent a primary element of the home environment. We hypothesized 
possible effects on parenting and the home environment to include: 

• opposition to work participation by partners which could increase conflict and possibly 
domestic violence; 

• reduced income from welfare and other benefits which may lead to increased incidence of 
neglect; 

• mandated participation in employment or educational activities may provide structure and 
routines to families; 

• mandated participation in employment or educational activities may affect parental depression 
or aggravation and stress in parenting, either positively or negatively, depending on the 
parents’ experience and success. 

We hypothesized that the home environment and parenting practices may in turn affect child 
outcomes in the following ways: 
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• children’s observation of domestic violence is associated with an increase in child behavior 
problems; 

• domestic violence between partners predicts child abuse which is strongly linked to behavior 
problems; 

• neglect has negative consequences for health, as well as cognitive and socioemotional 
development; 

• regular routines (adequate sleep, regular meals) benefit children; 

• aggravation and stress in parenting are related to child behavior problems. 

Possible Measures. Domestic Violence. After much discussion about issues of sensitivity of the 
topic and confidentiality, we decided that domestic violence measures could not be used over the 
phone. However, there are questions from the National Survey of Families and from the JOBS 
Evaluation that could be used in the home in a self-administered format. 

Additionally, Manpower Demonstration Research Corporation has developed a set of questions 
that specifically measures barriers to work which includes questions on domestic violence within 
the series of questions. These questions were asked in the JOBS Evaluation and can also be used 
in a self-administered format. 

Abuse/neglect: Links to administrative records may be the best way to get at information on 
child abuse and neglect. After much discussion about issues of confidentiality and the restricted 
population of people included in administrative data, the discussion moved in the direction of 
asking questions on a survey that would indicate individuals at risk for abusive behavior. There 
is a single question in the JOBS Evaluation that can be used as an indicator as well as questions 
from Abt Associates that can be used. In the end, it was decided that the combination of 
administrative data and a few survey questions would be most informative. 

We also briefly discussed the idea of using child abuse and neglect data as indicator data. Many 
states are already engaged in efforts to track child abuse and neglect over time. 

Routines. Data on family routines can not be collected through administrative data. For survey 
data, the technical assistance team suggested the Family Routines Inventory. The inventory was 
used in the Head Start/Public School Transition Survey. These questions ask about the 
frequency with which family members engage in individual and mutual activities and the 
regularity of daily schedules. The team also suggested particular consideration of a draft version 
of the same inventory from the Early Childhood Longitudinal Study that is currently being 
pretested over the phone. This is an even shorter scale, with only 7 items. 

Aggravation/stress in Parenting. Again, aggravation and stress in parenting are not available 
from administrative records, but there is a good measure for mixed mode surveys: the 5-item 
Aggravation in Parenting scale. This scale was developed for the JOBS evaluation and has 
strong psychometrics and predicts child behavior problems, based on analyses of the control 
group sample. 



II. CHILD OUTCOMES 



1. Child Health and Safety 

There are a number of ways that welfare reform may ultimately affect child health and safety. 
Some examples include: 

• sanctions may reduce income which could reduce money spent on food, which may leave the 
family short on food by the end of the month; 

• work requirements may lead the parent to enroll the child in a day care center where the child 
is given regular check ups and immunizations are made current. This may result in a better 
overall health rating for the child; 

• work requirements may lead to less parental supervision of the child which may result in an 
increase in number of accidents and injuries experienced by the child; or 

• family income may increase over time, leading to improvements in nutrition and safety. 

Possible Measures. The technical assistance team suggested a limited number of measures of 
hunger and nutrition. Steven Carlson of the Food and Consumer Service in the U.S. Department 
of Agriculture urged states to consider additional measures for this construct. He has 
subsequently provided Child Trends with sets of questions that ask the family about their food 
situation, which will be provided to states for their consideration. Immunization was discussed 
but it was agreed that survey measures of this construct are time consuming and problematic. 
Therefore states which are interested in examining policies related to immunization were urged 
to set up administrative systems to verify immunization. The group decided that immunization 
was not well suited for inclusion in a core set of constructs. One construct was added to the 
initial list and this was whether the child had a regular source of medical care. States also agreed 
to include teen childbearing as part of the common core. 

2. Education 

The education domain includes several constructs, some of which are more appropriate for some 
age groups than others. General hypotheses that might explain the relationship between welfare 
reform and different aspects of children’s education were presented: 

• parents who participate in work or training activities might establish regular routines for their 
children. The establishment of regular routines, such as nightly homework sessions, might 
have positive implications for children’s school progress and academic achievement; 

• welfare reform provisions, such as requiring that teen parents remain in school or 
implementing sanctions if children are not attending school, might increase parent’s 
monitoring of children’s school attendance. Regular school attendance might decrease the 
likelihood that children are held back a grade and may be positively associated with high 
school completion; 

• parents’ ability to monitor their children’s behaviors and assist with school work might be 
compromised because of increased demands on their time. Lack of parental monitoring may 
have negative implications for children’s school progress; 
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• family income may be increased due to employment or income disregards. Increases in family 
income may provide resources that facilitate learning; 

• welfare reform provisions’ work requirements are likely to lead to more children, particularly 
infants and toddlers, being placed in child care. Placement in child care of high quality may 
have positive implications for children’s school performance, while placement in low-quality 
care or inconsistent arrangements may undermine school performance; 

• welfare policies may bring about instability in living arrangements, such as family moves to 
find employment. Instability in living arrangements may have negative implications for 
children’s school progress. 

Possible Measures. Administrative Data . Some states may have access to standardized test 
scores and other data that they will be able to link to their research samples. However, gaining 
access to these data on an individual child level might prove to be very difficult. In addition, it is 
rare for schools to use the same tests, even within a particular county or school district. States 
need to explore the situation in their state. 

Survey Measures . Meeting participants were provided with several survey measures of different 
aspects of children’s education that can be administered over the phone or in an in-home 
interview. Sample size for a particular age group is a consideration for some of these measures, 
such as school readiness. 

School readiness (0-5). Child Trends provided everyone with a copy of the National Household 
Education Survey (NHES) School Readiness Interview which is a 12-item developmental profile 
for preschool-aged children. State participants voted to drop this construct from the core list 
because of small sample sizes for this age group. 

Engagement in school. States were provided with a measure of children’s engagement in school 
developed by Dr. James Connell. This measure is both reliable and valid, and is predictive of 
school outcomes, such as attendance, grade point average, test scores, suspensions, and grade 
retention. 

School attendance and grade repetition. Measuring school attendance and grade repetition is 
fairly straightforward, though it is possible that parents are not aware of all of their children’s 
absences. Items from the Prospects Survey and the National Household Education Survey were 
recommended to the states. 

Achievement. For states that are interested in conducting in-home interviews and who can then 
administer direct assessments of children’s cognitive achievement, Child Trends provided a 
table, compiled by Abt Associates, which presents a review of measures of children’s cognitive 
achievement and development. It was agreed that achievement tests would not be part of the 
common core. 

Graduation from high school. A question from National Education Longitudinal Study 1988 
Parent Questionnaire can be asked of parents. As with school readiness, this outcome only 
applies to a narrow age range. Therefore, it is necessary to examine the number of cases 
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available for analysis on this outcome. This construct was eliminated from the common core in 
the final discussion on Friday afternoon. 

Grades. State participants agreed that they would like to add a measure of children’s grades in 
school to the common core. 

3. Social and Emotional Adjustment 

Because of the number of different constructs under this domain, general hypotheses which may 
explain the relationship between welfare reform and children’s and adolescent’s social and 
emotional adjustment were presented: 

• state welfare policies may increase or decrease levels of maternal depression and stress. 
Changes in maternal depression rates or stress could have implications (either positive or 
negative) for children’s and adolescent’s mental health and behavior; 

• increased work requirements may lead to a decreased ability on the part of parents to monitor 
their children’s behavior. Lack of parental monitoring and supervision due to increased 
participation in the workforce may lead to an increase in problem behaviors; 

• welfare policies may bring about changes in family structure or family income. Changes in 
family structure or family income may have implications for children’s and adolescent’s 
mental health and behavior (either positive or negative); 

• child support enforcement policies may increase fathers’ involvement in their children’s lives. 
Increased father involvement may increase or decrease children’s behavior problems, 
depending on the quality and continuity of the relationship. 

Possible Measures. Administrative Data . There are several possibilities for collecting 
information about children’s social and emotional adjustment through administrative data. 
However, it was noted that these data will only be available for children who have entered the 
official system. For example, juvenile justice departments could provide data on official arrests 
and contacts with the police. However, these data do not capture those youth who have not been 
caught or who are engaging in behaviors that have not been serious enough to attract police 
attention. 

Survey Data . Meeting participants considered a number of primarily parent-report measures of 
children’s behaviors and emotional adjustment. For adolescents, it would be ideal to collect this 
information from the youth themselves. There are two possibilities. First, if conducting a 
telephone interview, it is possible to have parents "hand off' the phone to their adolescent child. 
This has been done previously in the National Commission on Children Telephone survey. For 
states who are considering in-home interviews, adolescents could answer questions using a self- 
administered questionnaire format. 

Behavior Problems. General behavior problems: Child Trends recommended the Behavior 
Problems Index (BPI) as a measure of general behavior problems. This is a 28-item parent- 
report rating scale of children’s behavior developed by Nick Zill and James Peterson, based on 
the Child Behavior Checklist developed by Achenbach and Edelbrock. There are versions 
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available for children ages 4-17. For shorter surveys, the technical assistance team recommended 
using two subscales from the BPI that tap externalizing and internalizing behaviors. 

Delinquency, drug and alcohol use. The notebooks that were distributed to meeting participants, 
included several questions that can be asked of parents about their adolescents’ behaviors. These 
come from the National Education Longitudinal Study (NELS) and the National Survey of 
America’s Families. Questions from the NELS 1988 about alcohol use, drug use and problem 
behaviors can also be asked of adolescents. However, states agreed that these should not be part 
of the common core. However, a single item concerning arrests was added to the common core. 

Positive Behaviors/Social Competence. Although measures of positive behaviors were not 
included in the measures notebook, state representatives agreed that these would be important to 
include as part of the common core. A measure that was used in the JOBS Evaluation and the 
New Chance Evaluation, the Positive Behaviors Scale, will be provided to states. 

Depression/Mental Health. For children and youth ages 2-17, a copy of a short mental health 
measure developed by Achenbach for the National Health Interview Survey (NHIS) was 
suggested. These items overlap with the Behavior Problems Index, as both measures were 
developed using items from the Child Behavior Checklist. The short NHIS scale is designed to 
measure mental health, whereas the BPI is a more general measure of behavior problems. 

For those states that are able to collect information about depressive symptoms from the 
adolescents themselves, a copy of the Center for Epidemiologic Studies Depression Scale for 
Children (CES-DC) was provided. This is a modified version of the Center for Epidemiologic 
Studies Depression Scale designed for use with adolescents ages 12-18. 



Children’s depression/mental health was eliminated from the common core. 

III. POSSIBILITIES FOR ADDITIONAL FUNDING 

Barbara Blum from the National Center for Children in Poverty introduced the panel of private 
foundation representatives. Jennifer Phillips, from the Charles Stewart Mott Foundation, 
provided a national foundation perspective about the importance of the Project on State-Level 
Child Outcomes. Judith Simpson, from the George Gund Foundation, provided a regional 
foundation perspective on how states should approach foundations to obtain funding to support 
their evaluations. Mary Bogle described an organization called Grant Makers for Children, 
Youth, and Families, a consortium of over 400 foundations. 

The National Perspective 

Jennifer Phillips presented three questions that foundations might ask themselves. First, why 
would they be interested in this type of research? Phillips noted that the end products of this 
research will be very important in shaping policy. Second, foundations might ask themselves 
what is the importance of measuring the effects of welfare reform on children? Third, what is 
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innovative about the Project on State-Level Child Outcomes, compared to other similar 
initiatives? Phillips noted that this project is one of the few forums where practitioners are 
formulating the research questions and the methods for collecting information on children. This 
project will form a "national patchwork" of information that as a whole, will provide important 
information on the well-being of children. This patchwork has the potential to reffame the 
current welfare debate. 

The Regional/Local Perspective 

Judith Simpson presented several techniques for approaching local and regional funders for 
support. First, she explained that there are two types of local and state funders: private 
foundations and community foundations. These foundations range from the very small to the 
very large. Most states have a regional association of grant makers. Simpson suggested that one 
way to approach these funders is to ask if the regional association would convene a meeting so 
that a state could provide an educational presentation about this project. 

Second, a welfare reform evaluation project needs to be framed appropriately for a state funder. 
Many foundations might say that they do not fund research, but this might actually mean that 
they do not have the staff to evaluate a research proposal. Instead, Simpson suggested to the 
states that they should say they would like to "engage in an assessment process" to determine 
how and in what ways welfare reforms are affecting children. 

Third, funders should not be overwhelmed with paper or descriptions of research methodologies 
or measures. Fourth, those who are approaching funders should acknowledge that this is a 
competitive environment. Foundations are asked not only to fund research but also are 
increasingly asked to "prop up" the safety net for poor families (for example, by funding food 
banks and homeless shelters). Funders will need to be shown that welfare reform research is 
critical in the context of other service delivery needs that they might be asked to fund. 

Simpson also noted that funders are interested in the availability of interim data. They do not 
necessarily want to wait five years for results. In addition, community foundations will want 
analyses for a particular community because they have geographic restrictions on funding. States 
who approach private funders should also note whether the foundation money will leverage other 
funds, either public or private. An additional "selling point" for the states that are participating in 
this project is that they have worked with the U.S. Department of Health and Human Services, 
the NICHD Family and Child Well-being Network, Child Trends, and all of the participating 
states to develop a common core of measures. 

Grant Makers for Children, Youth, and Families: 

Grant Makers for Children, Youth, and Families has over 400 foundations as members. This 
organization sponsors Learning Circles for these foundations. Three to five topics are 
highlighted each year. One of the Learning Circle topics this year is "Creative Foundation 
Responses to Devolution." Grant Makers is compiling a notebook that will contain information 
about various projects, including the Project on State-Level Child Outcomes. Mary Bogle asked 
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state representatives to let her know what successes they have had in working with private 
foundations. 

IV. NEXT STEPS FOR TECHNICAL ASSISTANCE 

It is anticipated that the group, potentially including all twelve states, will continue meeting over 
the next several years, as impact and indicator work is designed and implemented. Members of 
the technical assistance team agreed to be in regular contact with state participants about 
constructs and measures for their impact studies, as well as the indicators meeting. In addition, 
ways to share the information generated as a part of this process with states that are not involved 
will be considered. 

The core constructs chart that was distributed at the meeting needed to be revised, based on the 
discussions and recommendations of the meeting participants. A copy of the revised construct 
chart appears in the Appendix. 
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Summary of the Meeting to Inform the Child Care Components 
of State Welfare Waiver Evaluations 
February 7, 1997 



Key Findings, Measurement and Design Issues, and Recommendations 
for Addressing Child Care in the State Welfare Waiver Evaluations 



Martha J. Zaslow and Erin Oldham 
Child Trends, Inc. 
Washington, D.C. 



A meeting of child care researchers, federal and state child care administrators, and 
participants in the studies of child outcomes in the context of state welfare waiver evaluations 
was convened on February 7, 1997. The purpose of this meeting was to ask for input as to the 
most important aspects of child care to cover in the state welfare waiver evaluations, and for 
advice on specific measures of child care for inclusion in these evaluations. A second purpose of 
the meeting was to discuss more broadly the key issues concerning child care in the new policy 
context that should be examined in future research. 

This document summarizes (1) key findings from the research, as discussed at the meeting, 
that can help provide a context for the study of child care issues within the state welfare waiver 
evaluations; (2) design and measurement issues raised at the meeting that should be taken into 
account in the welfare waiver evaluation studies; (3) preliminary suggestions for what aspects of 
child care could be addressed in the evaluation studies, and what specific child care measures 
could be used in these studies. 



I. Kev Findings From the Child Care Research. 

Presented at the February 7th Meeting . 
that are Pertinent to the Welfare Waiver Evaluations 

We highlight here research findings presented at the February 7th meeting, that may be 
particularly important for understanding child care use and its implications in low income 
families. 

A. Quality of care and family income. Income is associated with quality in family day care 
homes as well as in child care centers. For families using family day care, the evidence indicates 
that quality of care increases with family income. For families using center care the relationship 
is more complex. Rather than children from families with the lowest income receiving the 
poorest quality care, it is children just above the poverty line who appear to be the most likely to 
get low quality care. 

B. Child care doesn’t only affect the child. Researchers at the meeting noted repeatedly that 
child care has the potential to affect not only children ’s development, but also mothers ’ ability to 
maintain employment and program participation, and family stress/stability. We need to 
consider simultaneously whether child care supports mothers’ employment and children’s 
development. 

C. Child care quality and children’s development. Studies consistently find that child care 
quality is related to child outcomes in multiple domains of development. 

A review of the evidence on how child care quality affects children’s development concurrently 
found that outcomes most commonly associated with higher quality care include: more optimal 
peer interactions, more positive social skills, a higher complexity of play, higher levels of social 
problem-solving, and fewer and less serious behavior problems. 

Patterns most consistently found in the research in association with lower quality care include 
children engaging more often in solitary play and wandering aimlessly, a lack of involvement in 
classroom activities, more crying and negativity, less sustained verbal interactions, and lower 
levels of language development. 

Longitudinal benefits (at preschool age) of higher quality of child care during the infancy period 
include less distractibility and more task-orientation, more considerate social interaction, better 
academic progress and fewer academic problems. 

While associations between child care quality and child outcomes are found quite consistently, in 
some research the magnitude of the relationship is modest. 

D. The importance of quality of child care for children in families at risk. Quality of care 
matters to children’s development across all income groups. However there is some evidence 
that the quality of care matters more for children from families at risk. For example, a study of 
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children in center care found that care quality had larger effects for children from families with 
lower parental education. 

E. The importance of aspects of child care to mothers’ ability to maintain employment and 
program participation. Mothers appear to be attuned to aspects of care that are particularly 
important to making their employment and program participation viable: stability/reliability, and 
convenience/location. 

F. Aspects of child care that are related to mothers’ discontinuing program participation. 

Mothers’ discontinuing their participation in a welfare-to-work program has been found to be 
related to concerns about the staff: child ratio, safety, and reliability of child care. 

G. Child care and family stress. Several researchers at the meeting noted evidence that a 
breakdown in child care arrangements acts as a source of stress to mothers and families. 
Consistent child care can stabilize a family otherwise characterized by turbulence, such as 
homelessness. 

H. Findings on family day care. Many low income families rely on family day care rather than 
formal care. A study of family day care found that much of that care is of poor quality. There is 
also evidence of much turnover of providers/ instability in family day care (noted by several 
researchers at the meeting). Findings on the quality of care in family day care settings noted at 
the meeting include the following points: 

1 . Quality of family day care was higher when the provider was licensed. 

2. The caregiver’s reasons for providing child care were important. The quality of care was 
higher when the caregiver reported that this was her chosen profession. It was lower when she 
noted that she worked primarily for the money, or that she cared for other children primarily as 
a way to stay with her own children. 

3. Quality of care was higher when the education of the provider was higher. 

4. Group size and ratio for older children did not seem to be very important in family day care, 
(though these are consistently found to be important in center day care), perhaps because 
group sizes in family day care are typically small and children are more likely to vary in age. 

5. However for infants, there seems to be an important distinction between care that is 
provided 1 : 1 (for example, a grandmother caring for one infant), and care that involves 2 or 
more children per caregiver. 

I. Findings on infant care. Studies are showing widespread use of child care for infants. The 
NICHD Study of Early Child Care has found that 72% of children in their sample had 
experienced child care (by a caregiver other than mother or father) by their first birthday, and that 
those children averaged 29 hours per week in care. Most of this care was in informal 
arrangements. With the number of babies in care so high, quality of care for infants is important. 
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1 . The particular features of care that matter to infants’ development vary depending on what 
aspect of development (for example, peer relations, attachment relationships with adults, 
curiosity) is being considered. However, overall, infants fare better when: 

group sizes were smaller and there were fewer children per adult, the physical setting was 
rated as safer and more stimulating, care givers had more education, care givers had less 
authoritarian attitudes about child rearing, and there was greater stability of care. 

2. But the importance of these different quality features for children’s development differed 
somewhat for care in a home as opposed to child care center. 

In home settings, going from one to two (or more) children per adult reduced quality, and 
authoritarian child rearing attitudes were associated with lower quality. 

In center care for infants, caregiver training and education, group size, and ratio were 
important. 

J. Findings on state child care regulations. A study carried out in Florida found that when 
state regulations changed, requiring ratios of 1 :4, instead of 1 :6, and increasing requirements for 
staff training, child outcomes improved. The study of Cost Quality and Child Outcomes in Child 
Care Centers also found higher quality center child care in states with tighter regulations. 

K. Findings on availability of care. Participants at the meeting pointed to evidence of 
substantial variation across communities in how many center slots there are for children. 

1 . Even across low income communities with similar demographic characteristics, the 
availability of center slots can differ dramatically. 

2. A new study is looking at how the supply and cost of care vary by community 
characteristics, such as the percent of children who are poor and the percent of female- headed 
households. Preliminary findings from this study will be available soon. 

3. At the meeting it was stressed that mothers’ choice of type of care is related to the 
availability of different types of care in a community. It is important to collect information 
about the type of care used by low income families, but this information needs to be placed in 
the context of care available within the community. 

L. Choice of type of care in relation to family race/ethnicity. There is evidence of differences 
in the type of care chosen by families of different race/ethnicity. Hispanic mothers are less likely 
to choose formal child care. This appears to be related both to the relative availability (lack of 
availability) of center care in neighborhoods with high concentrations of Hispanic families, but 
also to beliefs about child rearing. Latino mothers believe that in center care their children will 
become more independent and individualistic. This is not in accord with their socialization 
goals. In general, research needs to consider the issue of child care use (type and extent) in 
relation to culture and beliefs about child rearing. 
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M. Child health in center and family day care settings. The accumulated evidence is usually 
summarized as showing that children in center day care have more absences than children in 
family day care homes, who in turn have more absences than children cared for at home. 
However, when illnesses are examined instead of absences, children in day care homes exceed 
those in centers. 

Parent recall is an issue here. Children in centers are more likely to be excluded when they are 
ill. Parents with children in centers may be more likely to remember days involving absences. 

N. Use of child care subsidies. Findings show that a substantial proportion of families 
transitioning off of AFDC in the past were paying for child care, and not using subsidies This 
may reflect a lack of information about eligibility for subsidies. It may also reflect difficulty 
getting the subsidies. 



II. Design and Measurement Issues Raised at the Meeting 
Pertinent to the State Welfare Waiver Evaluations 



Participants at the meeting had been asked to comment on design and measurement issues 
pertinent to the state welfare evaluation studies. A summary of key issues raised at the meeting 
follows: 

A. Age of children focused on in the state studies 

1 . The role of child care will be different if the state evaluation is focusing on 5-12 year old 
children as opposed to younger children. The meeting did not focus specifically on after- 
school arrangements, care during the summer, self-care, or care during non-standard hours at 
length, but these may play a large role in the state studies. 

2. In states focusing on 5-12 year old children, we can still ask some key questions 
retrospectively, for example about stability of care. We cam also ask concurrently about after- 
school care, care during non-standard work hours, care used during the summer, and self-care. 

B. Timing of the survey in relation to random assignment 

1 . In states that choose to field a survey to augment their evaluations, the role of child care will 
also vary depending on the number of years after random assignment that the survey is 
administered. For example, the role of child care soon after random assignment may differ 
from the role it plays 3 or 4 years after random assignment. 

2. It is still important to study child care even 3-4 years after random assignment especially 
among respondents who are employed. It will still be important to know about type of care 
used, reliance on self-care by the child, handling of nontraditional hours and summer, and 
stability of care. 

C. Use of administrative data: challenges and opportunities 

We are at a turning point in terms of the use of administrative data to study child care issues in 
the states. 

1. A new set of projects launched by the Child Care Bureau involves collaborations by 
researchers, state child care administrators and resource and referral agencies to examine such 
issues as cost and availability of child care in relation to community characteristics. 
Preliminary findings from these studies will be available soon. 

2. It was noted at the meeting that there is drastic variation across locales in terms of how 
child care administrative data are collected and summarized, from pencil and paper tallies, to 
more sophisticated on-line systems. 
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3. In addition to variation in the quality of administrative data, participants at the meeting 
noted that these data are also generally available only for those receiving services and 
subsidies. Therefore, using only administrative data produces a biased sample, excluding 
those no longer receiving services and funds and those not currently utilizing services or 
subsidies that they are entitled to. A survey may be necessary to get information for a broader 
sample, including those who received services and subsidies in the past and those eligible but 
not receiving services or funds. 

4. There may be important changes in administrative data collection regarding child care in the 
future. While data were collected and summarized at the aggregate level in the past, states 
will now be append information that will make it possible to look at individual recipients, and 
relate child care use and use of subsidies to other data. 

5. We need to think about the extent to which these changes in administrative data collection 
can benefit the state evaluations, and the extent to which the changes will not be fully realized 
soon enough to build on in these evaluations but could be critical in further work. 

D. What can mothers report on? When should the state studies turn to other respondents 
and data collection strategies? 

1 . Concerns about maternal report. There was much discussion at the meeting about 
maternal report. There was agreement that mothers are good sources of information on some 
aspects of child care, but not very good sources at all regarding other aspects of child care. 

2. When are mothers good sources of information about child care? The researchers at the 
meeting felt that mothers could be turned to for information about the following aspects of 
child care. 

For a particular child (focal child): 

Number of different child care arrangements used regularly for focal child at present; type of 
care setting (center care, family day care, etc.) for primary arrangement or all arrangements; 
hours per week child participates; number of children present in group; ages of children 
present; number of care providers in group; relation of mother to care provider (is provider a 
relative or nonrelative); whether the care setting is licensed; how much household pays for 
primary arrangement or all arrangements for child; assistance in paying for child care; 
location of care/convenience of getting to care. 

For all children in the family: 

How many different care situations the mother relies on for all of her children currently; 
whether any one care setting provides care for more than one of her children; how much 
household pays for child care for all children in family; assistance in paying for child care 
for all children; whether any of children in family cares for self on own on a regular basis; 
whether any of children in family cares for younger siblings on a regular basis. 




43 



For all children or for focal child: 

How often a child care arrangement has broken down over the past (month, 6 months, year, 
other interval); number of different arrangements mother has had to rely on over the past 
(month, 6 months, year, other interval) for focal child or all children; whether mother has 
access to care when child or children are sick; how often mother has had to miss 
work/school/training because a child was sick and could not go to child care; how often 
mother has had to miss work/school/training because of any other child care issues; reliance 
on care for children during nontraditional hours; difficulty in arranging such care; if mother 
were entirely free to choose, would she change child care provider for focal child/any of her 
children; extent to which concerns about child care are a source of stress to mother; whether 
mother sometimes works hours that are not covered by child care arrangements for her 
child(ren). 

3. When is maternal report questionable? Mothers do not seem to be good reporters of 
caregiver education or training. 

In addition, it appears to be very important how questions about maternal satisfaction with 
child care are worded. When asked about their overall satisfaction with child care, mothers 
consistently report high levels of satisfaction, and their ratings do not correspond closely with 
on-site observations of the quality of care. It is possible that mothers react to global questions 
about satisfaction with child care on grounds that are different than what child care researchers 
are looking for when they are rating child care quality. In particular, mothers may complete 
such ratings from the perspective of the realistic constraints and demands of their lives (e.g., 
regarding employment, cost, other children’s needs). 

Yet when mothers are asked if they would choose to change child care providers if they were 
free to, mothers show much more variation in response. This may be a better way to approach 
the issue of maternal satisfaction with child care. 

4. What data could be obtained by contacting care providers? It was noted that maternal 
permission can be obtained to call or send a survey to the child’s child care provider. 

Providers can report on: 

Caregiver education and training; caregiver salaries; group size and ratio; ages of children; 
type of care; auspice of care; licensing and certification; hours and days care available; if in 
center, number of classrooms; ages of children who attend; proportion of children receiving 
subsidies; “intentionality,” or reasons provider is working in this capacity (as a profession, 
primarily to earn money, primarily as a means to be with her own children while earning 
money); authoritarian child rearing attitudes. 

5. What kinds of information require on-site observations? Certain aspects of quality 
require on-site direct observation for reliable measurement. These include observations of the 
quality and quantity of caregiver-child interaction, of child peer interactions, and of child task- 
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orientation (as opposed to aimless wandering). On-site observations can also yield 
information about the physical characteristics of the setting and safety. 

E. Care instability and the value of collecting data about child care quality in the context of 
the state welfare waiver evaluations 

An important concern was voiced about collecting data (via maternal report, provider report or 
direct observation) about a child’s current primary child care arrangement. There may be so 
much instability of care that attempting to measure the quality of care in any one setting may 
give us very little information about a particular child’s cumulative experiences. If we attempt 
to document child care experiences at only one point in time, we may have a very narrow 
window on child care quality, and a very limited basis for examining child care experience as a 
predictor of child outcomes. In short, it may be more valuable to view the child care 
information as one factor in the mother’s employment and well-being, than as a mediator of 
the child’s outcomes. 

Thus, the possibility was raised that it may be more important in this sample to obtain 
measures of: 

• convenience of care for parent 

• stability/continuity 

• cost 

• relationship with provider 

• degree to which care is a source of stress for the parent 

• mother’s perceptions of safety, reliability, trustworthiness of the provider 

We note that a one-time profile of the child’s current child care experience could serve as 
valuable descriptive information, helping us to document the kinds of child care children in 
families receiving or transitioning from public assistance experience. That is, even if we 
decide that a one-time measure of child care type and quality is of limited usefulness in 
predicting to child outcomes, we still may want to be able to describe the child care settings. In 
addition, it may be that arrangements are fairly similar, and that obtaining information on one 
setting reflects characteristics of other settings the child is in or has been in. 

F. Concerns about measures of caregiver sensitivity and the physical setting in direct 
observations of the quality of care 

It is important to be aware that many of the measures involving direct observation of the child 
care environment were designed using middle-class Caucasian samples. Within these 
measures of child care quality involving direct observation, the appropriateness of the ratings 
of caregiver sensitivity across different population groups has not been explicitly examined. 
“Sensitive” care giving may not look the same across ethnic groups; the cultural context of the 
interaction between caregiver and child is critical to understand. It is important to review 
ratings of caregiver sensitivity from this perspective. 
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In addition, in some currently used measures of the quality of the child care environment, 
ratings of caregiver-child interaction are made only if the ratings of the physical environment 
indicate a sufficient level of physical safety and presence of materials. Because caregiver-child 
interactions have an intrinsic value, and because the material focus of the measures might 
preclude ratings of caregiver-child interactions in child care arrangements with limited 
resources, it may be important in the future to eliminate the linkage between carrying out 
ratings of the physical environment and of caregiver-child interaction. 

G. Measures of care type used, and child outcome measures, must be understood in 

cultural context. 

Culture and beliefs can affect whether child care is used (see summary of findings). In 
addition, parent-report measures of child outcomes (for example, a description of the child’s 
behaviors) must also be seen in cultural context. For example, such measures might view 
independence and autonomy as positive, whereas these may go against socialization goals in 
some cultures. We need to be careful in our interpretation of parent-report child outcome 
measures. 
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III. Recommendations for Aspects of Care to Measure 
and for Specific Measures to Use in State Welfare Waiver Evaluations 



Researchers at the meeting stressed two main themes: the connection between child care and 
the mother’s ability to participate in work, and the connection between child care and child 
outcomes. Below we summarize aspects of child care that could be addressed in the state studies 
concerning each of these. We summarize aspects of care that could be addressed separately 
according to different options that states might take in collecting data: relying on mother as a 
respondent in a survey, obtaining permission to contact child care providers, and assessing child 
care quality through direct observation. 

A. Mother as respondent in a survey 

1 . States may want to know how child care affects the ability of recipients to get and maintain 
employment. Relevant aspects of child care include: 

a. the supply of child care (if through administrative data, then number of center slots and 
licensed family day care slots available per 1 ,000 children in recipients neighborhood) 

b. mother’s perception of difficulty in finding care for focal child/all children 

c. reliance on care for non-traditional hours; difficulty in finding such care 

d. the number of different arrangements the parent is having to use simultaneously for focal 
child/all children 

e. the cost of care for focal child/all children 

f. use of subsidies for care for focal child’s care/all children 

g. the convenience of care (location and transportation issues)— focal child/all children 

h. availability of care for child(ren) when ill 

I. frequency with which mother has had to miss work/school/training because of problems 
with child care (note nature of problem) 

j. frequency with which work schedules/ demands have disrupted child care arrangements 

k. rating of child care as a source of stress for the mother 

l. if mother were free to choose, would she change care provider for focal child/any of her 
children? 
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m. if mother has access to a phone at work, so that she can touch base with child care provider 
and/or children, and receive emergency calls while children are in child care 

2. States may want to know how child care or lack of child care affects the development of the 
child. In addition, (or instead), states may want to be able to describe the child care settings that 
children in the studies are experiencing. Here measures would pertain to the focal child, because 
we would be seeking to describe a particular child’s experiences and relate these experiences to 
his/her development. Relevant aspects of care to measure for these purposes include: 

a. type of care (primary arrangement/all current regular arrangements-- number of 
concurrent arrangements) 

b. hours per week in primary arrangement/all current regular arrangements 

c. number of children in child’s group 

d. number of providers in child’s group 

e. licensed care or not 

f. whether the child takes care of him/herself on an occasional or regular basis 

g. whether the child is cared for by a sibling on an occasional or regular basis 

h. how many caregivers child has had in the last year (or choose longer interval) 

3. For older children (7-14), it would be very important to go beyond consideration of center 
care, family day care, and reliance on relatives and neighbors as forms of care. Types of care for 
older children should include also after-school child care, after-school activities, boys and girls 
clubs, and regularly scheduled lessons. It would continue to be important to ask about self-care 
and care by siblings. It would also become important to ask about whether child is in charge of 
siblings or other children on a regular basis. 

B. Permission obtained to contact child care provider 

An intermediate method of getting reliable information on characteristics of care that are 
related to child care quality is to call the provider of child care, or send the provider a survey. In 
this case, states could address: 

a. number of children in child’s group 

b. number of caregivers in child’s group 

c. training and education of child’s primary provider 
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d. “intentionality” (reasons for being a child care provider) 

e. licensed or not 

f. auspice of care 

g. attitudes about caregiving (e.g., authoritarian) 

h. hours and days care provided 
I. cost of care 

C. Permission obtained to observe in care setting 



There is clear added value to doing direct observations of the physical environment in child 
care settings, and of interactive aspects of child care (caregiver-child interactions, and child’s 
interactions with peers). Additional recommendations regarding measures for direct 
observation of the child care setting would be made according to the specific hypotheses and 
research designs being used by states deciding to pursue this approach. 
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Figure 1 
Initial Model 
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Figure 4 



Conceptual Framework Following State Discussions 
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Figure 6 

Aspects of Constructs to be Measured Following State Prioritization 
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Figure 7 



Aspects of Child's Environment That May Be Affected bv Previous Columns 
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Aspects of Child Outcomes to be Measured Following State Prioritization 
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Prioritization of Outcomes for Welfare Reform Studies 
Project on State Level Child Outcomes 

AGE 0-5 

What are the child outcomes your state wants to measure? If you think any have been overlooked please 
add them in the blank table and list their priority. Please indicate C if you feel the construct should be 
core or included in all of the state evaluation or S if you feel the construct should be core only for your 
particular state. Please indicate a H, M, or L for high, medium, or low priority for use within an 
indicators project. 
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AGE 6-11 

Please indicate C if you feel the construct should be core or included in all of the state evaluation or S if you feel the construct 
should be core only for your particular state. Please indicate a H, M, or L for high, medium, or low priority for use within an 
indicators project. 



Health and Safety 


I 


I 


Education 


I 


I 


Social and Emotional 


I 


I 




m 


n 




m 


n 


Adjustment 


m 


n 




P 


d 




P 


d 




P 


d 




a 


I 




a 


I 




a 


I 




c 


c 




c 


c 




c 


c 




t 


a 




t 


a 




t 


a 






t 






t 






t 






0 






0 






0 






r 






r 






r 


Accidents and injuries 






Achievement tests 






Behavior problems 






Child abuse 






Dropping out 






Confidence/self-esteem/perceived 

self-competence 






Emergency room visits 






Educational expectations and 
aspirations 






Depression/mental health 






Hunger/nutrition 






Repeating a grade 






Drug/alcohol/tobacco use 






Immunization 






School attendance 






Fears, phobia, and anxiety 






Lead exposure 






School engagement (Scale 






Gang membership 






Limiting health (physical or 






measuring how much effort a 












mental) conditions 






child is putting into his/her 






Institutionalization (criminal, 












schoolwork) 






mental health) 






Morbidity (sickness/disease) 






School performance 






Juvenile justice/illegal activities 






Mortality 






School suspension/expulsion 






Life satisfaction 






Perception of safety 






Special education (referrals or 






Parent-child relationship 






Rating of child’s health (single 
question, parent rating) 






placements) 






Religiosity/spirituality 






Screens for developmental delay 
(e.g., did child walk on-time, 












Sibling relationships 






language development) 












Social skills 


















Teen pregnancy /abortion/child 
bearing 


















Volunteering 








56 



57 



AGE 12-17 

Please indicate C if you feel the construct should be core or included in all of the state evaluation or S if you feel the construct 
should be core only for your particular state. Please indicate a H, M, or L for high, medium, or low priority for use within an 
indicators project. 
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OTHER CONSTRUCTS 

List any additional constructs you feel should be measured along with the appropriate rating. 
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STATE: 



Prioritization of Intervening Mechanisms for Welfare Reform Studies 
Project on State Level Child Outcomes 



What are the intervening mechanisms your state wants to measure? If you think any have been 
overlooked please add them in the blank table and list their priority. Please indicate C if you feel the 
construct should be core or included in all of the state evaluations or S if you feel the construct should be 
core only for your particular state. Please indicate a H, M, or L for high, medium, or low priority for use 
within an indicators project. 
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Father/Absent 

Parent 

Characteristics and 
Involvement 


Impact 


Indicator 


Stability/ 

Turbulence 


Impact 


Indicator 


Use of Health and 
Human Services 


Impact 


Indicator 
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Stability of contact 
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Fchild Care 


Impact 
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Level of personal 
responsibility 

Problem solving 
skills 

Resident parent’s 
attitude/preference 
for work 

Socialization of the 
parent to work, 
routines, willing to 
stay employed 
(work ethic) 






Abuse-neglect 

Chores, housework 
by child 

Cognitive 

Stimulation 

Community 

involvement 

Discipline 

Drug-free, no 
alcohol abuse 

Harsh parenting 

Immunizations 

Parent-child 

interaction 

(warmth, 

aggravation) 

Parent’s 
mobilization of 
resources (car 
pools, sports, 
teams, free 
community 
activities) 

Parental monitoring 
(school on time, 
knowledge of 
friends) 

Parental school 
involvement 

Recreational time 
with children 

Regular routines 

Role modeling 
(work, education) 
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Social Support 


Impact 


Indicator 


Consumption 


Impact 


Indicator 


Resident 

Parent’s 

Psychological 

Well-being 

and Physical 

Health 


Impact 


Indicator 


Amount of social 






Distribution of 






Depression 






support 






income within 
the family 






Domestic 






Extended families & 






(how much is 






violence 






resources 






spent on 






/abusive 












whom) 






relationships 






Friendship Networks 






Housing 






Physical Health 






Kinship networks 






quality 






Self-efficacy/ 






Mother-figure, Father- 






Material 






Locus of control 






figure 






deprivation 






Self-esteem 






Organized Activities 






Neighborhood 












(community, church) 






quality 






Stress: 






Quality of social 






Resource 






degree and 






support 






utilization (% 






source (e.g., 












spent on child 






time, financial, 






Reciprocity of (give 






care, rent, 






parenting) 






vs. got) 






food) 












Satisfaction with 
social support 


















Type of social support 
(emotional, 
instrumental, social, 
parenting) 
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